Smoothly changing a focus of a camera between multiple target objects

ABSTRACT

Disclosed herein is a system to smoothly change the focus of a camera between multiple targets. The system can obtain an indication of a target, an indication of a manner of focus transition between a first target and a second target, and camera settings. The system can determine a point associated with the second target, where the point has a property that focusing the camera on the point places the second target in focus, and the point is closer to the current focus point of the camera than a substantial portion of other points having the property. The system can obtain a nonlinear function indicating a second manner of focus transition between the first target and the second target. The system can change the focus of the camera between the first target and the second target by changing the focus of the camera from the current focus point to the determined point based on the nonlinear function.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to the U.S. provisional patentapplication Ser. No. 63/222,287 filed Jul. 15, 2021 which isincorporated herein by reference in its entirety.

BACKGROUND

The current focusing systems for computer graphics (CG) cameras uselinear interpolation between positions of two target objects, namely,the first target object on which the camera is focused, and the secondtarget object to which the focus is transitioning. When the two targetobjects are far away, the focusing between the positions of the twotarget objects can produce popping in the focus, e.g., popping in thechange in blur of the elements in the image.

BRIEF DESCRIPTION OF THE DRAWINGS

Detailed descriptions of implementations of the present invention willbe illustrated and explained through the use of the accompanyingdrawings.

FIG. 1 shows changing a focus between two target objects in a sceneaccording to two different methods.

FIG. 2 shows a user interface to specify the target objects and a mannerof focus transition between the target objects.

FIG. 3 shows components of the system to produce a smooth focustransition between two target objects.

FIGS. 4A-4B show how to determine the new camera focus point.

FIGS. 5A-5D show lens profiles of various lenses.

FIG. 6 shows the nonlinear function that is an average of various lensprofiles.

FIG. 7 shows a data structure used to determine the new camera focuspoint.

FIG. 8 is a flowchart of a method to smoothly change a focus of a camerabetween multiple target objects.

FIG. 9 illustrates an example visual content generation system 900 asmight be used to generate imagery in the form of still images and/orvideo sequences of images.

FIG. 10 is a block diagram that illustrates a computer system 1000 uponwhich the computer systems of the systems described herein and/or visualcontent generation system 900 (see FIG. 9) may be implemented.

The technologies described herein will become more apparent to thoseskilled in the art from studying the Detailed Description in conjunctionwith the drawings. Embodiments or implementations describing aspects ofthe invention are illustrated by way of example, and the same referencescan indicate similar elements. While the drawings depict variousimplementations for the purpose of illustration, those skilled in theart will recognize that alternative implementations can be employedwithout departing from the principles of the present technologies.Accordingly, while specific implementations are shown in the drawings,the technology is amenable to various modifications.

DETAILED DESCRIPTION

Disclosed herein is a system and method to smoothly change the focus ofa computer graphics (CG) camera between multiple target objects. Thesystem can obtain an indication of each target object, such as thegeometry and the position of each target object. The system can obtainan indication of a manner of focus transition between a first targetobject and a second target object. The indication of the manner of focustransition can be a length of time to transition between the two targetsor a weight associated with each target position, which indicates thefocus position of the camera between the two target positions. Thesystem can obtain CG camera settings such as the current focus point ofthe CG camera, the position of the camera, and an indication of a depthof field of the CG camera. The CG camera is configured to transition thefocus from the first target object to the second target object. Thedepth of field is the distance between the nearest object and thefarthest object that are in focus in an image formed by the CG camera.The depth of field can depend on various camera settings such asaperture, current focus point, and focal length of the lens. The systemcan determine a second depth of field associated with the second targetobject, where the second depth of field indicates a near distance to theCG camera and a far distance to the CG camera, such that when a focuspoint of the CG camera is between the near distance and the fardistance, an object located between the near distance and the fardistance is in focus. The system can determine a point between the neardistance and the far distance closest to the current focus point of theCG camera. The system can obtain a nonlinear function indicating asmooth manner of focus transition between the first target object andthe second target object. The system can smoothly change the focus ofthe CG camera between the first target object and the second targetobject by changing the focus of the CG camera from the current focuspoint to the point within the second depth of field closest to thecurrent focus point. The system can smoothly change the focus based onthe nonlinear function.

The description and associated drawings are illustrative examples andare not to be construed as limiting. This disclosure provides certaindetails for a thorough understanding and enabling description of theseexamples. One skilled in the relevant technology will understand,however, that the invention can be practiced without many of thesedetails. Likewise, one skilled in the relevant technology willunderstand that the invention can include well-known structures orfeatures that are not shown or described in detail, to avoidunnecessarily obscuring the descriptions of examples.

Smoothly Change a Focus of a Camera Between Multiple Target Objects

FIG. 1 shows changing a focus between two target objects in a sceneaccording to two different methods. The row of images 100 shows a changein focus of a camera between the first target object (“first target”)110 and the second target object (“second target”) 120 over 24 frames ofa video or a film using a traditional system. The camera can be acomputer graphics (CG) camera, or can be a physical camera. Image 100-1shows the initial frame in the sequence where the CG camera istransitioning focus between first target object 110 and second targetobject 120. In the initial frame 100-1 the CG camera is focused on thefirst target object 110, with the second target object 120 blurred.Image 100-2 shows the sixth frame in the sequence. In the sixth frame100-2, the CG camera is already focused on the second target object 120,even though the transition of the focus should be completed by thetwenty-fourth frame, 100-3. As can be seen in row 100, the focaldifference, i.e., the blur, between the image 100-2 and the image 100-3is almost identical. Consequently, the quick change in focus betweenimages 100-1 and 100-2 results in a visual pop.

The row of images 130 shows a change in focus of the CG camera betweenthe first target object 110 and the second target object 120 over 24frames of video or film using the system described in this application.As can be seen in images 130-1, 130-2, 130-3, 130-4, the focus isgradually shifting from the first target object 110 to the second targetobject 120 over the 24 frames. In the initial frame 130-1, the firsttarget object 110 is in focus, in the in-between frames 130-2 and 130-3,both of the target objects 110, 120 have varying degrees of blur, and inthe final image 130-4, the focus has transitioned to the second targetobject 120. Consequently, the focal transition appears smooth in the rowof images 130.

Elements 140 and 150 show a focus chart indicating the sharpness ofcamera's focus. Element 140 indicates the sharpness of camera's focus atthe first target object 110, while element 150 indicates the sharpnessof cameras focus at the second target object 120.

FIG. 2 shows a user interface to specify the target objects and a mannerof focus transition between the target objects. User interface elements200, 210, 220 indicate the target objects, in this case Jane, boat, andJohn. User interface element 280 can indicate the current target objecton which a CG camera is focused.

User interface elements 230, 240 indicate a portion of the targetobjects 200, 210 on which a CG camera should focus. User interfaceelements 250, 260, 270 (only 3 labeled for brevity) indicate the mannerof focus transition between the target objects. For example, the mannerof focus transition can happen over half a second, 1 second, 1.5seconds. In addition, the user interface can receive an input from theuser specifying a different manner of transition, such as 2 seconds, 2.3seconds, etc. The system can take the new target object, such as targetobject 210, and the manner of focus transition, and produce an animationof a smooth focus transition between the old target object 200 and thenew target object 210.

FIG. 3 shows components of the system to produce a smooth focustransition between two target objects. The system 300 includes inputs310, a module 320, and outputs 330.

The inputs 310 include target positions 310-1, 310-2, target weights310-3, 310-4, camera position 310-5, and camera settings 310-6. Thetarget positions 310-1, 310-2 are positions in space of the targets 340,350. Initially, the camera 360 can be focused on the target 340, andwhen the focus changes finish, the camera 360 can be focused on thetarget 350. The target weights 310-3, 310-4 can be an indication of amanner of focus transition between the target 340 and the target 350.For example, the target weights 310-3, 310-4 can sum to one, and canindicate a linear transition of the camera focus between the targetpositions 310-1, 310-2, respectively. In other words, when the targetweight 310-3 is 1 and the target weight 310-4 is 0, the camera 360 isfocused on target position 310-1. When the target weight 310-3 is 0.5and the target weight 310-4 is 0.5, the camera 360 is focused on thepoint halfway between the target positions 310-1, 310-2. When the targetweight 310-3 is 0 and the target weight 310-4 is 1, the camera 360 isfocused on target position 310-2. As explained in FIG. 2 above, anotherway to indicate the manner of focus transition between the target 340and the target 350 is to specify the length of time over which the focustransition occurs.

The camera position 310-5 is the spatial position of camera 360. Thecamera settings 310-6 include a current focus point of the camera and anindication of a depth of field of the camera. The depth of field is adistance between a nearest object and a farthest object that are infocus in an image formed by the camera 360. The depth of field of thecamera 360 can depend on the aperture, focal length, and distance to thetarget. Therefore, the indication of the depth of field of the camera360 can include aperture, focal length, and distance to the target.

The module 320 can take the inputs 310 and can process the inputsthrough the submodules 320-1, 320-2, 320-3, and 320-4. The module 320can be a standalone software or a plug-in into a software system such asMaya, Unreal, Unity, MotionBuilder, etc. The submodule 320-1 cancalculate distances between the target positions 310-1, 310-2 and thecamera 360. The submodule 320-2 can determine a new focus point tominimize the distance to the current focus point, as explained in FIGS.4A-4B. The submodule 320-3 can convert the indication of the manner offocus transition between the targets 340, 350, such as target weights310-3, 310-4 or length of time, into a nonlinear function, as explainedin FIGS. 5A-5D and 6.

Finally, the submodule 320-4 can determine the focus position of thecamera 360 by interpolating between the current focus point and the newfocus point according to the nonlinear function. The submodule 320-4 canproduce the outputs 330 indicating the position of the camera focuspoint and a manner in which to transition the camera focus point. Forexample, the outputs 330 can include the position of the camera focuspoint and a time at which the camera's focus should move to the focuspoint.

The module 320 can receive an input from a user indicating a magnitudeof contribution of one or both of the submodules 320-2, 320-3. Forexample, the user can fully use the submodule 320-2, not use thesubmodule 320-2, or only partially use the submodule 320-2. The user canindependently control the magnitude of contribution of either submodule320-2 or 320-3. For example, the magnitude can vary between 0 and 1,where 0 indicates no contribution from the submodule and 1 indicatesfull contribution from the submodule.

In the case of the submodule 320-3, the magnitude indicates theinterpolation between the nonlinear function and a linear function. Forexample, if the user sets the magnitude of contribution of the submodule320-3 to 0.5, the nonlinear function can be interpolated halfway to alinear function. If the user sets the magnitude of contribution of thesubmodule 320-3 to 0, the nonlinear function becomes a linear function.If the user sets the magnitude of contribution of the submodule 320-3 to1, the nonlinear function is used.

In the case of the submodule 320-2, the magnitude indicates the size ofthe depth of field 415 in FIG. 4A surrounding the second target 450 inFIG. 4A. For example, if the magnitude of contribution of the submodule320-2 is set to 1, the depth of field 415 is unchanged. If the magnitudeof contribution of the submodule 320-2 is set to 0, the depth of fieldbecomes a bounding box of the second target 450. If the magnitude ofcontribution of the submodule 320-2 is set to 0.5, the depth of fieldbecomes a halfway interpolation between the depth of field 415 and thebounding box of the second target 450.

The key property of the system described in this application is thattargets 340, 350 must be and must stay sharp when focused upon. During atransition from target 340 to target 350 starting at time T₁ and endingat time T₂, target 340 is not sharp after T₁ and target 350 is sharp noearlier than T₂. There is no discontinuity in focus during thetransition. Additional submodules can be included in the module 320 thatfurther modify the behavior of the focus transition by, for example,introducing focus breathing. Focus breathing refers to the shifting ofeither the angle of view or the focal length of a lens when changing thefocus, depending on the definition. The two definitions are notequivalent: a lens with a constant focal length will exhibit narrowingof the angle of view at closer focus, and conversely, maintaining aconstant angle of view requires precise shortening of the focal lengthas focus is decreased.

FIGS. 4A-4B show how to determine the new camera focus point. In FIG.4A, the camera 400 is focused on target object 410. Even though thecamera 400 focuses on a current focus point 405, the whole region 440 isactually in focus. In addition, the longer the distance, d_(f), betweenthe position of the camera 400 and the current focus point 405, thebigger the region that is actually in focus. Specifically, a depth offield 420 associated with the target object 410 is bigger than a depthof field 415 associated with the target object 450 because the targetobject 410 is farther away from the camera 400 and the target object450.

For each target object 410, 450 a processor dynamically calculates thedepth of field 420, 415, respectively, so that when the camera focusesanywhere within the depth of field 420, 415, the target object 410, 450is in focus, respectively.

The target object 410 has the depth of field 420 which indicates all thepoints on which the camera 400 can be focused, while still keeping thetarget object 410 in focus. The camera focus point 405 is within thedepth of field 420. The depth of field 420 can be described using twodistances, a near distance 420-1 and a far distance 420-2. The camerafocus point 405 also has a depth of field 440 which indicates all thepoints which are in focus.

The camera 400 needs to transition the focus to the target object 450.To transition the focus to the target object 450, a hardware or asoftware processor executing instructions described in this applicationcan compute the substantially smallest distance 460 the camera focuspoint 405 needs to travel to put the target object 450 in focus. Tocompute the substantially smallest distance 460, the processor candetermine the depth of field 415 around the target object 450 andcompute the substantially closest point 480 within the depth of field415 to the current camera focus point 405. As seen in FIG. 4B, when thecurrent camera focus point 405 transitions to the camera focus point480, target object 450 is in focus and within the camera depth of field490, while the initial target object 410 is not in focus.

The labels in FIGS. 4A-4B and some additional labels are explainedbelow. Specifically:

-   -   hfd is Hyperfocal distance    -   dnear is Near distance    -   dfar is Far distance    -   f is Focal length    -   a is Aperture    -   c is Circle of confusion    -   d is Distance to target    -   df is Camera's focus distance    -   dA is Distance from camera to target A    -   dA,near is Closest focus distance that keeps target A in focus    -   dA,far is Farthest focus distance that keeps target A in focus.

Further, the relationship between above variables is defined as:hfd=f+(f*f)/(a*c)dnear=d*(hfd−f)/(hfd+(d−2*f))dfar=d*(hfd−f)/(hfd−d).

To get new df, take the sum of:

-   -   Closest to df (dA, dA,near, dA,far)*weight    -   Closest to df (dB, dB,near, dB,far)*weight.

The system minimizes focus point movement, resulting in smoother focuspull. In one example, the calculation of the focus point is may not beinter-frame dependent and does not require additional triggers/inputsabout focus direction. The system is easily extensible. For example, thetarget weights 310-3, 310-4 in FIG. 3 can be changed to be nonlinear.The submodules 320-2 and 320-3 in FIG. 3 can be easily and independentlydisabled.

The substantially closest point 480 within the depth of field 415 can beany point of 90% of the points closest to the current focus point 405.For example, the substantially closest point 480A can be behind thetarget object 450, thus causing the camera to overshoot the targetobject 450, and then settle the focus on the target object 450. Theovershooting of focus and refocusing on the target object 450 can createa desirable artistic effect.

FIGS. 5A-5D show lens profiles of various lenses. The graphs 500, 510,520, 530 show the physical lens profiles of various lenses. As can beseen in FIGS. 5A-5D the physical lens profiles are not linear. Thegraphs 500, 510, 520, 530 show the focus distance on the X axis versusthe lens ring rotation on the Y axis. The graphs 505, 515, 525, 535 arenonlinear functions approximating the graphs 500, 510, 520, 530. As canbe seen in FIGS. 5A-5D, the graphs 500, 510, 520, 530, when the lensring rotation is 0, small changes to the lens ring rotation cause largechanges in the focus distance. When the lens rotation is greater than 2,large changes to the lens ring rotation cause small changes in the focusdistance.

The nonlinear functions 505, 515, 525 show that the lens ring rotationis inversely proportional to the distance. The nonlinear function 535shows that the lens ring rotation is inversely proportional to thesquare root of the distance. As can be seen, the lens profiles varybased on the lens type.

FIG. 6 shows the nonlinear function that is an average of various lensprofiles. The nonlinear function 600 can be represented using theformula:

${Q = {( {1 - \sqrt{1 - \frac{1}{( {1 + d} )}}} )*10}},$

where Q is the lens ring rotation distance and d is the focus distancein meters. A processor can receive an indication of the lens type from acamera, or from a user. For example, if a CG camera needs to match thereal camera, once a lens is mounted on the real camera, the lens canautomatically communicate the type of lens to the CG camera. Based onthe type of lens, the processor can determine the appropriate nonlinearfunction 505, 515, 525, 535 in FIGS. 5A-5D.

In one example, based on the indication of the lens type, the processorcan select the appropriate nonlinear function 505, 515, 525, 535 inFIGS. 5A-5D, 600 that best matches the camera lens. In another example,the processor can automatically fit the appropriate nonlinear functionto the lens profile 500, 510, 520, 530. If no lens type is indicated,the processor can select the nonlinear function 600 that represents anaverage of the various lens profiles.

In addition, the lens profile 500, 510, 520, 530 can be representedusing a machine learning/artificial intelligence model (AI). In somecases, the lens profile 500, 510, 520, 530 cannot be modeled using thenonlinear function 505, 515, 525, 535, and AI can be used to determinethe manner of focus transition based on the lens profile 500, 510, 520,530. Further, the lens profile 500, 510, 520, 530 can change based onthe distance between the camera and the target object. Instead of usingthe same nonlinear function 505, 515, 525, 535 for various distancesbetween the camera and the target object, the AI can determine themanner of focus transition based on the distance between the camera andthe target object.

Further, the AI can anticipate a motion of a target object, and evenbefore the target object moves, the focus puller can begin the focustransition to the anticipated new position of the target object.

FIG. 7 shows a data structure used to determine the new camera focuspoint. For each target object 700 among multiple target objects, aprocessor can create a first distance 710, a second distance 720, and anexact distance 730, based on the position of the camera, the position ofthe object, and camera settings such as depth of field.

The first distance 710 and the second distance 720 indicate a depth offield associated with the target object 700, that is a region withinwhich the focus point of the camera can lie and have the target object700 in focus. The exact distance 730 indicates a distance between thecurrent focus point of the CG camera and the target object 700.

The processor can update which of the first distance 710, the seconddistance 720, and the exact distance 730 are closest to the currentfocus point of the camera. For example, the processor can rank thedistances 710, 720, 730 as shown in column 740, where rank of 1indicates that the first distance 710 is the closest to the currentfocus point of the camera.

FIG. 8 is a flowchart of a method to smoothly change a focus of a camerabetween multiple target objects. The camera can be a CG camera or can bea physical camera. In step 800, a processor can obtain an indication ofeach target object among the multiple target objects, an indication of afirst manner of focus transition between a first target object and asecond target object among the multiple target objects, and the CGcamera setting. The manner of focus transition can be a speed of focustransition and can be indicated in various ways such as by a length oftime to transition between two targets, by linear weights indicating thefocus position of the camera, by a curve indicating the position of thecamera, etc. The CG camera is configured to transition the focus fromthe first target object to the second target object. The CG camerasetting can include a current focus point of the CG camera, a region ofacceptable focus, position of the camera, aperture of the camera, etc.The region of acceptable focus can be an indication of a depth of fieldof the camera.

In step 810, the processor can determine a point associated with thesecond target object, where the point has a property that focusing theCG camera on the point causes the second target object to be in focus.The point associated with the second target object is closer to thecurrent focus point of the CG camera than a substantial portion of otherpoints having the property. For example, the point can be the closestpoint to the current focus point. In another example, the point can becloser to the current focus point than 50% of the points having theproperty. In another example, the point can be closer to the currentfocus point than 10% of the points having the property.

To determine the point associated with the second target object, theprocessor can determine a depth of field associated with the secondtarget object. The depth of field is a region between a nearest objectand a farthest object that are in focus in an image formed by the CGcamera. The depth of field indicates a near distance to the CG cameraand a far distance to the CG camera. When a focus point of the CG camerais between the near distance and the far distance, an object locatedbetween the near distance and the far distance is in focus. Theprocessor can determine the point between the near distance and the fardistance substantially closest to the current focus point of the CGcamera. Substantially closest can include points that are closer to thecurrent focus point than 10% of the points within the depth of field.

In step 820, the processor can obtain a nonlinear function indicating asecond manner of focus transition between the first target object andthe second target object, as described in FIGS. 5A-5D and 6. Thenonlinear function can include one or more nonlinear functionsdescribing the manner of focus transition. For example, the nonlinearfunction can model a physical lens as shown in FIGS. 5A-5D. Thenonlinear function can include an indication of an animation associatedwith the second manner of focus transition between the first targetobject and the second target object. For example, the nonlinear functioncan include a time dependent curve indicating speed of focus transition.The nonlinear functions can be preset. The processor can present theuser with an option of one or more nonlinear functions to select. Forexample, the preset nonlinear functions can include an abrupt focustransition, smooth focus transition, focus transition to overshoot thetarget and pulled back to focus on the target, etc. Various nonlinearfunctions can be combined with each other.

The processor can adjust a magnitude of influence of steps 810 and 820based on, for example, user input, as described in relation tosubmodules 320-2, 320-3 in FIG. 3, and as described below. For example,effectively, the processor can avoid performing the step 810 and/or step820.

In step 830, the processor can change the focus of the CG camera betweenthe first target object and the second target object by changing thefocus of the CG camera from the current focus point to the pointsubstantially closest to the current focus point (“point”) based on thenonlinear function.

To ensure the chosen distance option remains consistent during focustransition, the processor can maintain a data structure associated witheach target object. For each target object that is not being focused toor focused away from, the processor can create the data structurecontaining a first distance, a second distance, and an exact distance,where the first distance and the second distance indicate region withinwhich a focus point of the CG camera can lie and have each target objectin focus, for example, the depth of field. The calculation of the first,second, and exact distances is based on the position of the camera, theposition of the object, and camera settings such as depth of field. Theexact distance indicates a distance between the current focus point ofthe CG camera and each target object. The values need to be continuallyrecalculated because the camera and targets are moving. The processorcan update which of the first distance, the second distance, and theexact distance is closest to the current focus point of the CG camera.Subsequently, if the camera focus is shifting to a new target object,the processor can simply determine the new camera focus point byretrieving the closest point among the first, second, and exactdistances from the data structure.

To determine the nonlinear function to use, the processor can obtain anindication of a physical camera lens. The processor can retrieve thenonlinear function representing focus behavior of the physical cameralens from a database. The processor can use the nonlinear function tochange the focus of the CG camera.

The processor can adjust the magnitude of contribution of the point tothe manner of focus transition as described in relation to the submodule320-2 in FIG. 3. The processor can receive an input indicating amagnitude of contribution of the point. The magnitude can indicate tofully use the point, to not use the point, or to partially use thepoint. The processor can change the focus of the CG camera between thefirst target object and the second target object by changing the focusof the CG camera from the current focus point to the point associatedwith the second target object, based on the nonlinear function and themagnitude, as described in FIG. 3 in relation to the submodule 320-2.

The processor can adjust the magnitude of contribution of the nonlinearfunction to the manner of focus transition, as described in relation tothe submodule 320-3 in FIG. 3. The processor can receive a user inputindicating a magnitude of contribution of the nonlinear function. Themagnitude can indicate to fully use the nonlinear function, to not usethe nonlinear function, or to partially use the nonlinear function. Theprocessor can change the focus of the CG camera between the first targetobject and the second target object by changing the focus of the CGcamera from the current focus point to the point associated with thesecond target object based on the magnitude of contribution of thenonlinear function.

To model the manner of focus transition of a physical lens, theprocessor can obtain the nonlinear function substantially inverselyproportional to a distance between the CG camera and the current focuspoint and indicating the second manner of focus transition between thefirst target object and the second target object. Alternatively, theprocessor can obtain the nonlinear function inversely proportional to asquare root of the distance between the CG camera and the current focuspoint and indicating the second manner of focus transition between thefirst target object and the second target object.

Visual Content Generation System

FIG. 9 illustrates an example visual content generation system 900 asmight be used to generate imagery in the form of still images and/orvideo sequences of images. Visual content generation system 900 mightgenerate imagery of live action scenes, computer-generated scenes, or acombination thereof. In a practical system, users are provided withtools that allow them to specify, at high levels and low levels wherenecessary, what is to go into that imagery. For example, a user might bean animation artist and might use visual content generation system 900to capture interaction between two human actors performing live on asound stage, replace one of the human actors with a computer-generatedanthropomorphic non-human being that behaves in ways that mimic thereplaced human actor's movements and mannerisms, and then add in acomputer-generated third character and background scene elements, all inorder to tell a desired story or generate desired imagery.

Still images that are output by visual content generation system 900might be represented in computer memory as pixel arrays, such as atwo-dimensional array of pixel color values, each associated with apixel having a position in a two-dimensional image array. Pixel colorvalues might be represented by three or more (or fewer) color values perpixel, such as a red value, a green value, and a blue value (e.g., inRGB format). Dimensions of such a two-dimensional array of pixel colorvalues might correspond to a preferred and/or standard display scheme,such as 1920-pixel columns by 1280-pixel rows or 4096-pixel columns by2160-pixel rows, or some other resolution. Images might or might not bestored in a certain structured format, but either way, a desired imagemay be represented as a two-dimensional array of pixel color values. Inanother variation, images are represented by a pair of stereo images forthree-dimensional presentations and in other variations, an imageoutput, or a portion thereof, might represent three-dimensional imageryinstead of just two-dimensional views. In yet other embodiments, pixelvalues are data structures and a pixel value can be associated with apixel and can be a scalar value, a vector, or another data structureassociated with a corresponding pixel. That pixel value might or mightnot include color values, and might include depth values, alpha values,weight values, object identifiers, or other pixel value components.

A stored video sequence might include a plurality of images such as thestill images described above, but where each image of the plurality ofimages has a place in a timing sequence and the stored video sequence isarranged so that when each image is displayed in order, at a timeindicated by the timing sequence, the display presents what appears tobe moving and/or changing imagery. In one representation, each image ofthe plurality of images is a video frame having a specified frame numberthat corresponds to an amount of time that would elapse from when avideo sequence begins playing until that specified frame is displayed. Aframe rate might be used to describe how many frames of the stored videosequence are displayed per unit time. Example video sequences mightinclude 24 frames per second (24 FPS), 50 FPS, 140 FPS, or other framerates. In some embodiments, frames are interlaced or otherwise presentedfor display, but for clarity of description, in some examples, it isassumed that a video frame has one specified display time, but othervariations might be contemplated.

One method of creating a video sequence is to simply use a video camerato record a live action scene, i.e., events that physically occur andcan be recorded by a video camera. The events being recorded can beevents to be interpreted as viewed (such as seeing two human actors talkto each other) and/or can include events to be interpreted differentlydue to clever camera operations (such as moving actors about a stage tomake one appear larger than the other despite the actors actually beingof similar build, or using miniature objects with other miniatureobjects so as to be interpreted as a scene containing life-sizeobjects).

Creating video sequences for storytelling or other purposes often callsfor scenes that cannot be created with live actors, such as a talkingtree, an anthropomorphic object, space battles, and the like. Such videosequences might be generated computationally rather than capturing rightfrom live scenes. In some instances, an entirety of a video sequencemight be generated computationally, as in the case of acomputer-animated feature film. In some video sequences, it is desirableto have some computer-generated imagery and some live action, perhapswith some careful merging of the two.

While computer-generated imagery might be creatable by manuallyspecifying each color value for each pixel in each frame, this is likelytoo tedious to be practical. As a result, a creator uses various toolsto specify the imagery at a higher level. As an example, an artist mightspecify the positions in a scene space, such as a three-dimensionalcoordinate system, of objects and/or lighting, as well as a cameraviewpoint, and a camera view plane. From that, a rendering engine couldtake all of those as inputs, and compute each of the pixel color valuesin each of the frames. In another example, an artist specifies positionand movement of an articulated object having some specified texturerather than specifying the color of each pixel representing thatarticulated object in each frame.

In a specific example, a rendering engine performs ray tracing wherein apixel color value is determined by computing which objects lie along aray traced in the scene space from the camera viewpoint through a pointor portion of the camera view plane that corresponds to that pixel. Forexample, a camera view plane might be represented as a rectangle havinga position in the scene space that is divided into a grid correspondingto the pixels of the ultimate image to be generated, and if a raydefined by the camera viewpoint in the scene space and a given pixel inthat grid first intersects a solid, opaque, blue object, that givenpixel is assigned the color blue. Of course, for moderncomputer-generated imagery, determining pixel colors—and therebygenerating imagery—can be more complicated, as there are lightingissues, reflections, interpolations, and other considerations.

As illustrated in FIG. 9, a live action capture system 902 captures alive scene that plays out on a stage 904. Live action capture system 902is described herein in greater detail, but might include computerprocessing capabilities, image processing capabilities, one or moreprocessors, program code storage for storing program instructionsexecutable by the one or more processors, as well as user input devicesand user output devices, not all of which are shown.

In a specific live action capture system, cameras 906(1) and 906(2)capture the scene, while in some systems, there might be other sensor(s)908 that capture information from the live scene (e.g., infraredcameras, infrared sensors, motion capture (“mo-cap”) detectors, etc.).On stage 904, there might be human actors, animal actors, inanimateobjects, background objects, and possibly an object such as a greenscreen 910 that is designed to be captured in a live scene recording insuch a way that it is easily overlaid with computer-generated imagery.Stage 904 might also contain objects that serve as fiducials, such asfiducials 912(1)-(3), that might be used post-capture to determine wherean object was during capture. A live action scene might be illuminatedby one or more lights, such as an overhead light 914.

During or following the capture of a live action scene, live actioncapture system 902 might output live action footage to a live actionfootage storage 920. A live action processing system 922 might processlive action footage to generate data about that live action footage andstore that data into a live action metadata storage 924. Live actionprocessing system 922 might include computer processing capabilities,image processing capabilities, one or more processors, program codestorage for storing program instructions executable by the one or moreprocessors, as well as user input devices and user output devices, notall of which are shown. Live action processing system 922 might processlive action footage to determine boundaries of objects in a frame ormultiple frames, determine locations of objects in a live action scene,where a camera was relative to some action, distances between movingobjects and fiducials, etc. Where elements have sensors attached to themor are detected, the metadata might include location, color, andintensity of overhead light 914, as that might be useful inpost-processing to match computer-generated lighting on objects that arecomputer-generated and overlaid on the live action footage. Live actionprocessing system 922 might operate autonomously, perhaps based onpredetermined program instructions, to generate and output the liveaction metadata upon receiving and inputting the live action footage.The live action footage can be camera-captured data as well as data fromother sensors.

An animation creation system 930 is another part of visual contentgeneration system 900. Animation creation system 930 might includecomputer processing capabilities, image processing capabilities, one ormore processors, program code storage for storing program instructionsexecutable by the one or more processors, as well as user input devicesand user output devices, not all of which are shown. Animation creationsystem 930 might be used by animation artists, managers, and others tospecify details, perhaps programmatically and/or interactively, ofimagery to be generated. From user input and data from a database orother data source, indicated as a data store 932, animation creationsystem 930 might generate and output data representing objects (e.g., ahorse, a human, a ball, a teapot, a cloud, a light source, a texture,etc.) to an object storage 934, generate and output data representing ascene into a scene description storage 936, and/or generate and outputdata representing animation sequences to an animation sequence storage938.

Scene data might indicate locations of objects and other visualelements, values of their parameters, lighting, camera location, cameraview plane, and other details that a rendering engine 950 might use torender computer-generated imagery (CGI). For example, scene data mightinclude the locations of several articulated characters, backgroundobjects, lighting, etc., specified in a two-dimensional space,three-dimensional space, or other dimensional space (such as a2.5-dimensional space, three-quarter dimensions, pseudo-3D spaces, etc.)along with locations of a camera viewpoint and view plane from which torender imagery. For example, scene data might indicate that there is tobe a red, fuzzy, talking dog in the right half of a video and astationary tree in the left half of the video, all illuminated by abright point light source that is above and behind the camera viewpoint.In some cases, the camera viewpoint is not explicit, but can bedetermined from a viewing frustum. In the case of imagery that is to berendered to a rectangular view, the frustum would be a truncatedpyramid. Other shapes for a rendered view are possible and the cameraview plane could be different for different shapes.

Animation creation system 930 might be interactive, allowing a user toread in animation sequences, scene descriptions, object details, etc.,and edit those, possibly returning them to storage to update or replaceexisting data. As an example, an operator might read in objects fromobject storage into a baking processor 942 that would transform thoseobjects into simpler forms and return those to object storage 934 as newor different objects. For example, an operator might read in an objectthat has dozens of specified parameters (movable joints, color options,textures, etc.), select some values for those parameters, and then savea baked object that is a simplified object with now fixed values forthose parameters.

Rather than requiring user specification of each detail of a scene, datafrom data store 932 might be used to drive object presentation. Forexample, if an artist is creating an animation of a spaceship passingover the surface of the Earth, instead of manually drawing or specifyinga coastline, the artist might specify that animation creation system 930is to read data from data store 932 in a file containing coordinates ofEarth coastlines and generate background elements of a scene using thatcoastline data.

Animation sequence data might be in the form of time series of data forcontrol points of an object that has attributes that are controllable.For example, an object might be a humanoid character with limbs andjoints that are movable in manners similar to typical human movements.An artist can specify an animation sequence at a high level, such as“the left hand moves from location (X1, Y1, Z1) to (X2, Y2, Z2) overtime T1 to T2,” at a lower level (e.g., “move the elbow joint 2.5degrees per frame”), or even at a very high level (e.g., “character Ashould move, consistent with the laws of physics that are given for thisscene, from point P1 to point P2 along a specified path”).

Animation sequences in an animated scene might be specified by whathappens in a live action scene. An animation driver generator 944 mightread in live action metadata, such as data representing movements andpositions of body parts of a live actor during a live action scene.Animation driver generator 944 might generate corresponding animationparameters to be stored in animation sequence storage 938 for use inanimating a CGI object. This can be useful where a live action scene ofa human actor is captured while wearing mo-cap fiducials (e.g.,high-contrast markers outside actor clothing, high-visibility paint onactor skin, face, etc.) and the movement of those fiducials isdetermined by live action processing system 922. Animation drivergenerator 944 might convert that movement data into specifications ofhow joints of an articulated CGI character are to move over time.

Rendering engine 950 can read in animation sequences, scenedescriptions, and object details, as well as rendering engine controlinputs, such as a resolution selection and a set of renderingparameters. Resolution selection might be useful for an operator tocontrol a trade-off between speed of rendering and clarity of detail, asspeed might be more important than clarity for a movie maker to testsome interaction or direction, while clarity might be more importantthan speed for a movie maker to generate data that will be used forfinal prints of feature films to be distributed. Rendering engine 950might include computer processing capabilities, image processingcapabilities, one or more processors, program code storage for storingprogram instructions executable by the one or more processors, as wellas user input devices and user output devices, not all of which areshown.

Visual content generation system 900 can also include a merging system960 that merges live footage with animated content. The live footagemight be obtained and input by reading from live action footage storage920 to obtain live action footage, by reading from live action metadatastorage 924 to obtain details such as presumed segmentation in capturedimages segmenting objects in a live action scene from their background(perhaps aided by the fact that green screen 910 was part of the liveaction scene), and by obtaining CGI imagery from rendering engine 950.

Merging system 960 might also read data from rule sets formerging/combining storage 962. A very simple example of a rule in a ruleset might be “obtain a full image including a two-dimensional pixelarray from live footage, obtain a full image including a two-dimensionalpixel array from rendering engine 950, and output an image where eachpixel is a corresponding pixel from rendering engine 950 when thecorresponding pixel in the live footage is a specific color of green,otherwise output a pixel value from the corresponding pixel in the livefootage.”

Merging system 960 might include computer processing capabilities, imageprocessing capabilities, one or more processors, program code storagefor storing program instructions executable by the one or moreprocessors, as well as user input devices and user output devices, notall of which are shown. Merging system 960 might operate autonomously,following programming instructions, or might have a user interface orprogrammatic interface over which an operator can control a mergingprocess. In some embodiments, an operator can specify parameter valuesto use in a merging process and/or might specify specific tweaks to bemade to an output of merging system 960, such as modifying boundaries ofsegmented objects, inserting blurs to smooth out imperfections, oradding other effects. Based on its inputs, merging system 960 can outputan image to be stored in a static image storage 970 and/or a sequence ofimages in the form of video to be stored in an animated/combined videostorage 972.

Thus, as described, visual content generation system 900 can be used togenerate video that combines live action with computer-generatedanimation using various components and tools, some of which aredescribed in more detail herein. While visual content generation system900 might be useful for such combinations, with suitable settings, itcan be used for outputting entirely live action footage or entirely CGIsequences. The code may also be provided and/or carried by a transitorycomputer-readable medium, e.g., a transmission medium such as in theform of a signal transmitted over a network.

According to one embodiment, the techniques described herein areimplemented by one or more generalized computer systems programmed toperform the techniques pursuant to program instructions in firmware,memory, other storage, or a combination. Special-purpose computingdevices may be used, such as desktop computer systems, portable computersystems, handheld devices, networking devices, or any other device thatincorporates hardwired and/or program logic to implement the techniques.

One embodiment might include a carrier medium carrying image data orother data having details generated using the methods described herein.The carrier medium can comprise any medium suitable for carrying theimage data or other data, including a storage medium, e.g., solid-statememory, an optical disk, or a magnetic disk, or a transient medium,e.g., a signal carrying the image data such as a signal transmitted overa network, a digital signal, a radio frequency signal, an acousticsignal, an optical signal, or an electrical signal.

Computer System

FIG. 10 is a block diagram that illustrates a computer system 1000 uponwhich the computer systems of the systems described herein and/or visualcontent generation system 900 (see FIG. 9) may be implemented. Computersystem 1000 includes a bus 1002 or other communication mechanism forcommunicating information, and a processor 1004 coupled with bus 1002for processing information. Processor 1004 may be, for example, ageneral-purpose microprocessor.

Computer system 1000 also includes a main memory 1006, such as arandom-access memory (RAM) or other dynamic storage device, coupled tobus 1002 for storing information and instructions to be executed byprocessor 1004. Main memory 1006 may also be used for storing temporaryvariables or other intermediate information during execution ofinstructions to be executed by processor 1004. Such instructions, whenstored in nontransitory storage media accessible to processor 1004,render computer system 1000 into a special-purpose machine that iscustomized to perform the operations specified in the instructions.

Computer system 1000 further includes a read only memory (ROM) 1008 orother static storage device coupled to bus 1002 for storing staticinformation and instructions for processor 1004. A storage device 1010,such as a magnetic disk or optical disk, is provided and coupled to bus1002 for storing information and instructions.

Computer system 1000 may be coupled via bus 1002 to a display 1012, suchas a computer monitor, for displaying information to a computer user. Aninput device 1014, including alphanumeric and other keys, is coupled tobus 1002 for communicating information and command selections toprocessor 1004. Another type of user input device is a cursor control1016, such as a mouse, a trackball, or cursor direction keys forcommunicating direction information and command selections to processor1004 and for controlling cursor movement on display 1012. This inputdevice typically has two degrees of freedom in two axes, a first axis(e.g., x) and a second axis (e.g., y), allowing the device to specifypositions in a plane.

Computer system 1000 may implement the techniques described herein usingcustomized hardwired logic, one or more ASICs or FPGAs, firmware and/orprogram logic which in combination with the computer system causes orprograms computer system 1000 to be a special-purpose machine. Accordingto one embodiment, the techniques herein are performed by computersystem 1000 in response to processor 1004 executing one or moresequences of one or more instructions contained in main memory 1006.Such instructions may be read into main memory 1006 from another storagemedium, such as storage device 1010. Execution of the sequences ofinstructions contained in main memory 1006 causes processor 1004 toperform the process steps described herein. In alternative embodiments,hardwired circuitry may be used in place of or in combination withsoftware instructions.

The term “storage media” as used herein refers to any nontransitorymedia that store data and/or instructions that cause a machine tooperate in a specific fashion. Such storage media may includenonvolatile media and/or volatile media. Nonvolatile media includes, forexample, optical or magnetic disks, such as storage device 1010.Volatile media includes dynamic memory, such as main memory 1006. Commonforms of storage media include, for example, a floppy disk, a flexibledisk, a hard disk, a solid state drive, magnetic tape, or any othermagnetic data storage medium, a CD-ROM, any other optical data storagemedium, any physical medium with patterns of holes, a RAM, a PROM, anEPROM, a FLASH-EPROM, NVRAM, any other memory chip or cartridge.

Storage media is distinct from but may be used in conjunction withtransmission media. Transmission media participates in transferringinformation between storage media. For example, transmission mediaincludes coaxial cables, copper wire, and fiber optics, including thewires that include bus 1002. Transmission media can also take the formof acoustic or light waves, such as those generated during radio-waveand infrared data communications.

Various forms of media may be involved in carrying one or more sequencesof one or more instructions to processor 1004 for execution. Forexample, the instructions may initially be carried on a magnetic disk orsolid-state drive of a remote computer. The remote computer can load theinstructions into its dynamic memory and send the instructions over anetwork connection. A modem or network interface local to computersystem 1000 can receive the data. Bus 1002 carries the data to mainmemory 1006, from which processor 1004 retrieves and executes theinstructions. The instructions received by main memory 1006 mayoptionally be stored on storage device 1010 either before or afterexecution by processor 1004.

Computer system 1000 also includes a communication interface 1018coupled to bus 1002. Communication interface 1018 provides a two-waydata communication coupling to a network link 1020 that is connected toa local network 1022. For example, communication interface 1018 may be anetwork card, a modem, a cable modem, or a satellite modem to provide adata communication connection to a corresponding type of telephone lineor communications line. Wireless links may also be implemented. In anysuch implementation, communication interface 1018 sends and receiveselectrical, electromagnetic, or optical signals that carry digital datastreams representing various types of information.

Network link 1020 typically provides data communication through one ormore networks to other data devices. For example, network link 1020 mayprovide a connection through local network 1022 to a host computer 1024or to data equipment operated by an Internet Service Provider (ISP)1026. ISP 1026 in turn provides data communication services through theworldwide packet data communication network now commonly referred to asthe “Internet” 1028. Local network 1022 and Internet 1028 both useelectrical, electromagnetic, or optical signals that carry digital datastreams. The signals through the various networks and the signals onnetwork link 1020 and through communication interface 1018, which carrythe digital data to and from computer system 1000, are example forms oftransmission media.

Computer system 1000 can send messages and receive data, includingprogram code, through the network(s), network link 1020, andcommunication interface 1018. In the Internet example, a server 1030might transmit a requested code for an application program through theInternet 1028, ISP 1026, local network 1022, and communication interface1018. The received code may be executed by processor 1004 as it isreceived, and/or stored in storage device 1010 or other nonvolatilestorage for later execution.

Operations of processes described herein can be performed in anysuitable order unless otherwise indicated herein or otherwise clearlycontradicted by context. Processes described herein (or variationsand/or combinations thereof) may be performed under the control of oneor more computer systems configured with executable instructions and maybe implemented as code (e.g., executable instructions, one or morecomputer programs, or one or more applications) executing collectivelyon one or more processors, by hardware or combinations thereof. The codemay be stored on a computer-readable storage medium, for example, in theform of a computer program comprising a plurality of instructionsexecutable by one or more processors. The computer-readable storagemedium may be nontransitory. The code may also be provided carried by atransitory computer-readable medium, e.g., a transmission medium such asin the form of a signal transmitted over a network.

Conjunctive language, such as phrases of the form “at least one of A, B,and C,” or “at least one of A, B and C,” unless specifically statedotherwise or otherwise clearly contradicted by context, is otherwiseunderstood with the context as used in general to present that an item,term, etc., may be either A or B or C, or any nonempty subset of the setof A and B and C. For instance, in the illustrative example of a sethaving three members, the conjunctive phrases “at least one of A, B, andC” and “at least one of A, B and C” refer to any of the following sets:{A}, {B}, {C}, {A, B}, {A, C}, {B, C}, {A, B, C}. Thus, such conjunctivelanguage is not generally intended to imply that certain embodimentsrequire at least one of A, at least one of B, and at least one of C eachto be present.

The use of examples, or exemplary language (e.g., “such as”) providedherein, is intended merely to better illuminate embodiments of theinvention and does not pose a limitation on the scope of the inventionunless otherwise claimed. No language in the specification should beconstrued as indicating any non-claimed element as essential to thepractice of the invention.

In the foregoing specification, embodiments of the invention have beendescribed with reference to numerous specific details that may vary fromimplementation to implementation. The specification and drawings are,accordingly, to be regarded in an illustrative rather than a restrictivesense. The sole and exclusive indicator of the scope of the invention,and what is intended by the applicants to be the scope of the invention,is the literal and equivalent scope of the set of claims that issue fromthis application, in the specific form in which such claims issue,including any subsequent correction.

Further embodiments can be envisioned to one of ordinary skill in theart after reading this disclosure. In other embodiments, combinations orsubcombinations of the above-disclosed invention can be advantageouslymade. The example arrangements of components are shown for purposes ofillustration and combinations, additions, rearrangements, and the likeare contemplated in alternative embodiments of the present invention.Thus, while the invention has been described with respect to exemplaryembodiments, one skilled in the art will recognize that numerousmodifications are possible.

For example, the processes described herein may be implemented usinghardware components, software components, and/or any combinationthereof. The specification and drawings are, accordingly, to be regardedin an illustrative rather than a restrictive sense. It will, however, beevident that various modifications and changes may be made thereuntowithout departing from the broader spirit and scope of the invention asset forth in the claims and that the invention is intended to cover allmodifications and equivalents within the scope of the following claims.

All references, including publications, patent applications, andpatents, cited herein are hereby incorporated by reference to the sameextent as if each reference were individually and specifically indicatedto be incorporated by reference and were set forth in its entiretyherein.

Remarks

The terms “example,” “embodiment,” and “implementation” are usedinterchangeably. For example, references to “one example” or “anexample” in the disclosure can be, but not necessarily are, referencesto the same implementation; and, such references mean at least one ofthe implementations. The appearances of the phrase “in one example” arenot necessarily all referring to the same example, nor are separate oralternative examples mutually exclusive of other examples. A feature,structure, or characteristic described in connection with an example canbe included in another example of the disclosure. Moreover, variousfeatures are described which can be exhibited by some examples and notby others. Similarly, various requirements are described which can berequirements for some examples but no other examples.

The terminology used herein should be interpreted in its broadestreasonable manner, even though it is being used in conjunction withcertain specific examples of the invention. The terms used in thedisclosure generally have their ordinary meanings in the relevanttechnical art, within the context of the disclosure, and in the specificcontext where each term is used. A recital of alternative language orsynonyms does not exclude the use of other synonyms. Specialsignificance should not be placed upon whether or not a term iselaborated or discussed herein. The use of highlighting has no influenceon the scope and meaning of a term. Further, it will be appreciated thatthe same thing can be said in more than one way.

Unless the context clearly requires otherwise, throughout thedescription and the claims, the words “comprise,” “comprising,” and thelike are to be construed in an inclusive sense, as opposed to anexclusive or exhaustive sense; that is to say, in the sense of“including, but not limited to.” As used herein, the terms “connected,”“coupled,” or any variant thereof means any connection or coupling,either direct or indirect, between two or more elements; the coupling orconnection between the elements can be physical, logical, or acombination thereof. Additionally, the words “herein,” “above,” “below,”and words of similar import can refer to this application as a whole andnot to any particular portions of this application. Where contextpermits, words in the above Detailed Description using the singular orplural number may also include the plural or singular numberrespectively. The word “or” in reference to a list of two or more itemscovers all of the following interpretations of the word: any of theitems in the list, all of the items in the list, and any combination ofthe items in the list. The term “module” refers broadly to softwarecomponents, firmware components, and/or hardware components.

While specific examples of technology are described above forillustrative purposes, various equivalent modifications are possiblewithin the scope of the invention, as those skilled in the relevant artwill recognize. For example, while processes or blocks are presented ina given order, alternative implementations can perform routines havingsteps, or employ systems having blocks, in a different order, and someprocesses or blocks may be deleted, moved, added, subdivided, combined,and/or modified to provide alternative or subcombinations. Each of theseprocesses or blocks can be implemented in a variety of different ways.Also, while processes or blocks are at times shown as being performed inseries, these processes or blocks can instead be performed orimplemented in parallel, or can be performed at different times.Further, any specific numbers noted herein are only examples such thatalternative implementations can employ differing values or ranges.

Details of the disclosed implementations can vary considerably inspecific implementations while still being encompassed by the disclosedteachings. As noted above, particular terminology used when describingfeatures or aspects of the invention should not be taken to imply thatthe terminology is being redefined herein to be restricted to anyspecific characteristics, features, or aspects of the invention withwhich that terminology is associated. In general, the terms used in thefollowing claims should not be construed to limit the invention to thespecific examples disclosed herein, unless the above DetailedDescription explicitly defines such terms. Accordingly, the actual scopeof the invention encompasses not only the disclosed examples, but alsoall equivalent ways of practicing or implementing the invention underthe claims. Some alternative implementations can include additionalelements to those implementations described above or include fewerelements.

Any patents and applications and other references noted above, and anythat may be listed in accompanying filing papers, are incorporatedherein by reference in their entireties, except for any subject matterdisclaimers or disavowals, and except to the extent that theincorporated material is inconsistent with the express disclosureherein, in which case the language in this disclosure controls. Aspectsof the invention can be modified to employ the systems, functions, andconcepts of the various references described above to provide yetfurther implementations of the invention.

To reduce the number of claims, certain implementations are presentedbelow in certain claim forms, but the applicant contemplates variousaspects of an invention in other forms. For example, aspects of a claimcan be recited in a means-plus-function form or in other forms, such asbeing embodied in a computer-readable medium. A claim intended to beinterpreted as a means-plus-function claim will use the words “meansfor.” However, the use of the term “for” in any other context is notintended to invoke a similar interpretation. The applicant reserves theright to pursue such additional claim forms in either this applicationor in a continuing application.

The invention claimed is:
 1. A method to change a focus of a computergraphics (CG) camera between multiple target objects, the methodcomprising: obtaining an indication of a first target object and asecond target object among the multiple target objects, an indication ofa first manner of focus transition between the first target object andthe second target object among the multiple target objects, and the CGcamera settings, wherein the CG camera is configured to transition thefocus from the first target object to the second target object, whereinthe CG camera settings include a current focus point of the CG camera;obtaining a nonlinear function indicating a second manner of focustransition between the first target object and the second target object,wherein obtaining the nonlinear function includes: obtaining anindication of a physical camera lens; retrieving the nonlinear functionrepresenting focus behavior of the physical camera lens; and changingthe focus of the CG camera between the first target object and thesecond target object based on the nonlinear function.
 2. The method ofclaim 1, comprising: determining a point associated with the secondtarget object, wherein the point has a property that focusing the CGcamera on the point causes the second target object to be in focus,wherein the point associated with the second target object is closer tothe current focus point of the CG camera than a substantial portion ofother points having the property.
 3. The method of claim 2, whereindetermining the point associated with the second target objectcomprises: determining a region of acceptable focus associated with thesecond target object, wherein the region of acceptable focus is a regionbetween a nearest object and a farthest object that are in focus in animage formed by the CG camera, wherein the region of acceptable focusindicates a near distance to the CG camera and a far distance to the CGcamera, wherein when a focus point of the CG camera is between the neardistance and the far distance, an object located between the neardistance and the far distance is in focus; and determining the pointbetween the near distance and the far distance substantially closest tothe current focus point of the CG camera.
 4. The method of claim 2,comprising: for a target object among the multiple target objects:creating a first distance, a second distance, and an exact distance,wherein the first distance and the second distance indicate regionwithin which a focus point of the CG camera can lie and have the targetobject in focus, wherein the exact distance indicates a distance betweenthe current focus point of the CG camera and the target object; andupdating which of the first distance, the second distance, and the exactdistance are closest to the current focus point of the CG camera.
 5. Themethod of claim 2, comprising: receiving an input indicating a magnitudeof contribution of the point to changing the focus of the CG camera,wherein the magnitude can indicate to partially use the point; andchanging the focus of the CG camera between the first target object andthe second target object by changing the focus of the CG camera from thecurrent focus point to the point associated with the second targetobject, based on the nonlinear function and the magnitude.
 6. The methodof claim 1, comprising: receiving a user input indicating a magnitude ofcontribution of the nonlinear function to changing the focus of the CGcamera, wherein the magnitude can indicate to partially use thenonlinear function; and changing the focus of the CG camera between thefirst target object and the second target object based on the magnitudeof contribution of the nonlinear function.
 7. The method of claim 1,wherein the nonlinear function comprises an indication of an animationassociated with the second manner of focus transition between the firsttarget object and the second target object.
 8. The method of claim 1,comprising: obtaining the nonlinear function substantially inverselyproportional to a distance between the CG camera and the current focuspoint and indicating the second manner of focus transition between thefirst target object and the second target object.
 9. The method of claim1, comprising: obtaining the nonlinear function substantially inverselyproportional to a square root of a distance between the CG camera andthe current focus point and indicating the second manner of focustransition between the first target object and the second target object.10. At least one computer-readable storage medium carrying instructions,which, when executed by at least one data processor of a system, causethe system to: obtain an indication of a first target object and asecond target object, an indication of a first manner of focustransition between the first target object and the second target object,and camera settings, wherein a camera is configured to transition afocus from the first target object to the second target object, whereinthe camera settings include a current focus point of the camera; obtaina nonlinear function indicating a second manner of focus transitionbetween the first target object and the second target object, by:obtaining the nonlinear function substantially inversely proportional toa distance between the camera and the current focus point and indicatingthe second manner of focus transition between the first target objectand the second target object; and change the focus of the camera betweenthe first target object and the second target object based on thenonlinear function.
 11. The computer-readable storage medium of claim10, causing the system to: determine a point associated with the secondtarget object, wherein the point has a property that focusing the cameraon the point causes the second target object to be in focus, wherein thepoint associated with the second target object is closer to the currentfocus point of the camera than a substantial portion of other pointshaving the property.
 12. The computer-readable storage medium of claim11, causing the system to determine the point associated with the secondtarget object by: determining a region of acceptable focus associatedwith the second target object, wherein the region of acceptable focus isa region between a nearest object and a farthest object that are infocus in an image formed by the camera, wherein the region of acceptablefocus indicates a near distance to the camera and a far distance to thecamera, wherein when a focus point of the camera is between the neardistance and the far distance, an object located between the neardistance and the far distance is in focus; and determining the pointbetween the near distance and the far distance substantially closest tothe current focus point of the camera.
 13. The computer-readable storagemedium of claim 11, causing the system to: for a target object: create afirst distance, a second distance, and an exact distance, wherein thefirst distance and the second distance indicate region within which afocus point of the camera can lie and have the target object in focus,wherein the exact distance indicates a distance between the currentfocus point of the camera and the target object; and update which of thefirst distance, the second distance, and the exact distance are closestto the current focus point of the camera.
 14. The computer-readablestorage medium of claim 11, causing the system to: receive an inputindicating a magnitude of contribution of the point to changing thefocus of the camera, wherein the magnitude can indicate to partially usethe point; and change the focus of the camera between the first targetobject and the second target object by changing the focus of the camerafrom the current focus point to the point associated with the secondtarget object, based on the nonlinear function and the magnitude. 15.The computer-readable storage medium of claim 10, causing the system to:obtain an indication of a physical camera lens; retrieve the nonlinearfunction representing focus behavior of the physical camera lens; anduse the nonlinear function to change the focus of the camera.
 16. Thecomputer-readable storage medium of claim 10, causing the system to:receive a user input indicating a magnitude of contribution of thenonlinear function to changing the focus of the camera, wherein themagnitude can indicate to partially use the nonlinear function; andchange the focus of the camera between the first target object and thesecond target object based on the magnitude of contribution of thenonlinear function.
 17. The computer-readable storage medium of claim10, wherein the nonlinear function comprises an indication of ananimation associated with the second manner of focus transition betweenthe first target object and the second target object.
 18. Thecomputer-readable storage medium of claim 10, causing the system to:obtain the nonlinear function substantially inversely proportional to asquare root of the distance between the camera and the current focuspoint and indicating the second manner of focus transition between thefirst target object and the second target object.
 19. A systemcomprising: at least one hardware processor; and at least onenon-transitory memory storing instructions, which, when executed by theat least one hardware processor, cause the system to: obtain anindication of a first target object and a second target object, anindication of a first manner of focus transition between the firsttarget object and the second target object, and camera settings, whereina camera is configured to transition a focus from the first targetobject to the second target object, wherein the camera settings includea current focus point of the camera; obtain a nonlinear functionindicating a second manner of focus transition between the first targetobject and the second target object; receive a user input indicating amagnitude of contribution of the nonlinear function to changing thefocus of the camera, wherein the magnitude can indicate to partially usethe nonlinear function; and change the focus of the camera between thefirst target object and the second target object based on the nonlinearfunction and the magnitude of contribution of the nonlinear function.20. The system of claim 19, comprising instructions to: determine apoint associated with the second target object, wherein the point has aproperty that focusing the camera on the point causes the second targetobject to be in focus, wherein the point associated with the secondtarget object is closer to the current focus point of the camera than asubstantial portion of other points having the property.
 21. The systemof claim 20, comprising instructions to determine the point associatedwith the second target object by: determining a region of acceptablefocus associated with the second target object, wherein the region ofacceptable focus is a region between a nearest object and a farthestobject that are in focus in an image formed by the camera, wherein theregion of acceptable focus indicates a near distance to the camera and afar distance to the camera, wherein when a focus point of the camera isbetween the near distance and the far distance, an object locatedbetween the near distance and the far distance is in focus; anddetermining the point between the near distance and the far distancesubstantially closest to the current focus point of the camera.
 22. Thesystem of claim 20, comprising instructions to: for a target object:create a first distance, a second distance, and an exact distance,wherein the first distance and the second distance indicate regionwithin which a focus point of the camera can lie and have the targetobject in focus, wherein the exact distance indicates a distance betweenthe current focus point of the camera and the target object; and updatewhich of the first distance, the second distance, and the exact distanceare closest to the current focus point of the camera.
 23. The system ofclaim 20, comprising instructions to: receive an input indicating amagnitude of contribution of the point to changing the focus of thecamera, wherein the magnitude of contribution of the point can indicateto partially use the point; and change the focus of the camera betweenthe first target object and the second target object by changing thefocus of the camera from the current focus point to the point associatedwith the second target object, based on the nonlinear function and themagnitude of contribution of the point.
 24. The system of claim 19,comprising instructions to: obtain an indication of a physical cameralens; retrieve the nonlinear function representing focus behavior of thephysical camera lens; and use the nonlinear function to change the focusof the camera.
 25. The system of claim 19, wherein the nonlinearfunction comprises an indication of an animation associated with thesecond manner of focus transition between the first target object andthe second target object.
 26. The system of claim 19, comprisinginstructions to: obtain the nonlinear function substantially inverselyproportional to a distance between the camera and the current focuspoint and indicating the second manner of focus transition between thefirst target object and the second target object.
 27. The system ofclaim 19, comprising instructions to: obtain the nonlinear functionsubstantially inversely proportional to a square root of a distancebetween the camera and the current focus point and indicating the secondmanner of focus transition between the first target object and thesecond target object.